Papers with cross-lingual transfer tasks
Isomorphic Transfer of Syntactic Structures in Cross-Lingual NLP (P18-1)
Copied to clipboard
| Challenge: | Using frameworks such as Universal Dependencies (UD) to transfer knowledge between languages can be challenging because of variation in syntactic structures. |
| Approach: | They propose a typologically driven method which reduces anisomorphism in UD treebanks by considering both morphological and structural properties. |
| Outcome: | The proposed method is effective for machine translation and cross-lingual sentence similarity. |
InteMATs: Integrating Granularity-Specific Multilingual Adapters for Cross-Lingual Transfer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work relies on full-model fine-tuning on large parallel datasets to enhance cross-lingual alignment of MLLMs. |
| Approach: | They propose an approach that integrates multilingual adapters trained on texts of different levels of granularity into multilingual models. |
| Outcome: | The proposed approach improves the performance of multilingual language models on low-resource languages. |
mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for improving multilingual models only use entity information in pretraining and do not explicitly use entities in downstream tasks. |
| Approach: | They propose to leverage Wikipedia entity representations for downstream tasks . they train a multilingual language model with 24 languages with entity representation . |
| Outcome: | The proposed model outperforms word-based models in cross-lingual transfer tasks. |
Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent mPLMs have shown impressive performance on crosslingual transfer tasks . however, the performance is often hindered when a lowresource target language is written in a different script than the high-resource source language. |
| Approach: | They propose a transliteration-based method to improve cross-lingual alignment between languages using diverse scripts. |
| Outcome: | The proposed method outperforms the original model on Englishcentric transfer tasks up to 50%. |
Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)
Copied to clipboard
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks. |
| Approach: | They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains . |
| Outcome: | The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks. |